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ABSTRACT 



This document describes a study that designed, 



developed, and evaluated the Pennsylvania school-district database 
program for use by educational decision makers. The database contains 
current information developed from data provided by the Pennsylvania 
Department of Education and describes each of the 500 active school 
districts in the state. PEP PC was designed to enable easy 
accessibility of data for viewing and comparing school district 
information. A preliminary version of PEP PC was tested through a 
series of user tryouts, conducted at the individual, small group, and 
site levels. Data were collected through interviews, observations, a 
user's survey, and users' post-tryout software evaluations. Findings 
of the user's survey indicate that the microcomputer operating system 
used by PEP PC, the IBM compatible system, was available to 89 
percent of district administrators • The field tests showed that 
district administrators preferred a microcomputer database program 
that is user-friendly. The availability of district information can 
provide the basis for comparing the conditions of school districts 
and act as a powerful tool for district assessment, planning, and 
decision making. Three tables and three figures are included. 
Appendices contain copies of the database evaluation form and user's 
survey. (LMl) 
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Abstract 



The purpose of this study was to design, develop, and evaluate the 
Pennsylvania school district database program for use by educational decisionmakers. 
The database contains current information developed from data provided by the 
Pennsylvania Department of Education and describes each of the 500 active school 
districts in Pennsylvania. PEPS PC was designed to enable easy accessibility of data 
for viewing and comparing school district information. 

This software program was developed within the framework of a formative 
evaluation study. The project plan was adapted from the Bertram and Childers (1974) 
model for evaluation of educational products and consisted of the following three 
components: (a) project planning strategy, (b) design and development strategy, and 
(c) program diffusion strategy. The emphasis of this study was on design and 
development through a series of user tryouts. 

Using a preliminary version of PEPS PC, the program was tested with a series 
of one-to-one tryouts, small group tryouts, and on-site tryouts. The system was 
revised and again tested with user tryouts. Data were collected by evaluator 
interviews and observations during tryouts, a user's survey of the target population, 
and a software evaluation form, completed by users after tryouts. 

The main result of the user's survey was that the microcomputer operating 
system used by PEPS PC, the IBM compatible system, was available to 89% of district 
administrators. The field testing indicated that school district administrators prefer a 
microcomputer database program that is user friendly and includes: (a) clear prompts 
and screen displays, (b) frequent use of HELP screens, and (c) relatively quick 
operation time. The use of options that permit the use of queries and provide a means 
of comparing districts are desirable. 

The availability of easily accessed school district information can provide the 
foundation for feedback regarding the condition of school districts with respect to 
other districts and provide district leaders with a powerful tool for district assessment, 
negotiations, planning, and decision making. 
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The Development and Evaluation of a 
User-Friendly Database for Describing PA's School Districts 



The public school system has changed greatly in the last several years with 
respect to public school administra'ion. The number and kinds of educational 
professionals has increased, teachers' unions have become more powerful, and 
scarcity of public funds has forced stricter auditing procedures (Cunningham, 1982; 
Humphries, 1986). The public continues to demand measures of accountability and 
assurances that the schools are succeeding (Herman & Winters, 1989). As the 
participants in the school system increase, the school superintendent's job becomes 
more and more complex (Humphries, 1986). The superintendent must be able to 
communicate the condition of the schools to others, and through dialogue with 
participants, must be able to learn about what is happening in the schools and 
throughout the school district. 

Richard Wallace, (1985) former superintendent of Pittsburgh Public Schools, 
emphasizes the importance of information for educational leaders by saying "The first 
step in providing instructional leadership is to gain an understanding of the present 
state of the district. It is imperative that the superintendent analyze all relevant data 
at his/her disposal that might provide insights about the current functioning of schools 
in the district" (p. 9). 

This is not a simple task for today's educational administrator. The amount of 
information generated by educational organizations today has dramatically increased 
over the past several years (Burbach, 1989). Public schools, themselves, generate an 
enormous amount of data. They have student and personnel data, achievement data 
from local and state tests, and fiscal data for budgeting and accounting to name just 



a portion. In addition, local, state, and federal governments and private agencies are 
collecting data relevant to education (Burstein, 1984). The way schools access and 
use these data can affect administrative decisionmaking (Rhodes, 1988). 

The purpose of this study was to provide a means to access and browse 
information describing Pennsylvania's public school districts. This school district 
database was developed as a user-friendly, microcomputer system targeted for use by 
educational policymakers and decisionmakers for the purpose of providing a timely and 
relevant database to help support planning and decisionmaking. The use of a user- 
friendly, microcomputer system to access this database can provide a means to (a) 
quickly obtain useful data, (b) share information across different levels of the 
organization, and (c) use data to support decisions for planning and management 
(Cooley, 1990; Crawford, 1988; & Turbin, Gisher, & Altman, 1988). By providing a 
relevant and easily accessed database, and presenting information in a form that 
decisionmakers understand, information is more likely to be useful and used (Cooley, 
1983; Schellenberg, 1985). The use of relevant data that are timely and easily 
obtained can improve the decisionmaking and planning process, provide the 
opportunity to consider alternatives, and aid in the management of resources 
(Cunningham, 1982; Turbin et al., 1988). This paper will describe the development 
and evaluation of this database application. 

Program Description 
The PEPS PC school district database was developed in the context of the 
Pennsylvania Educational Policy Studies Project (PEPS) directed by Dr. William W. 
Cooley and located at the Learning Reseaich and Development Center at the University 
of Pittsburgh. An objective of the PEPS Project is to support the efforts of 
policymakers and school district administrators by providing a means of accessing 
relevant information that could support decisionmaking (Cooley, 1992). This 
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Pennsylvania school district database was developed as one way to achieve this 
objective. 

The design was based on an earlier version developed by William Cooley and 
James Bernauer. This current system, PEPS PC, is a stand-alone application designed 
for use on a DOS based, IBM compatible microcomputer system. This system is a 
menu-driven, user-friendly, microcomputer application designed to provide a means of 
selecting and manipulating the data so that school districts can be compared to other 
districts that are similar. The intent was not to interpret these data, but to display 
descriptive information that is relevant to educational policymaking and 
decisionmaking, and in a form that is easily understood. The following are a few 
examples of the way it may be used. School districts may be compared on the basis 
of their: 

a) instructional expenditures with other districts having similar available 
local revenue. 

b) student test results with other districts with a similar enrollment and 
student population. 

c) professional personnel salaries with other districts that are similar. 

The database used in this system was developed from data provided by the 
Pennsylvania Department of Education (PDE) to the PEPS Project. This file contains 
data collected for operational purposes by the various bureaus of the Department of 
Education (Cooley, 1990). A small part of these data is used to create the PEPS PC 
database which describes Pennsylvania public school districts. Although the 
information in the database will always be at least one year behind, Cooley (1990) 
reports that variations in ranking and comparisons between school districts remain 
relatively constant across years. This means, that despite the change in total values, 
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the relativity of the values remains stable, thus the database can remain useful as a 
means of comparing school districts. 

The database includes 128 variables that describe school districts. These 
variables are grouped as: (a) identification, (b) student, (c) revenues, (d) expenditures, 
and (e) professional personnel The identification variables relate to location, 
population density, as well as othfcr identifying information. The student variables are 
derived from the results of the TELLS test, a state mandated test administered in the 
spring of every school year in grades three, five, and eight in every school district in 
Pennsylvania (Hertzog, Masters, Miller, Simanovich, & Skiffington, 1985). The 
professional personnel variables are grouped by positions and include administrators, 
coordinators, and classroom teachers. Student and professional personnel data are 
summarized for each district. 

The financial variables include both revenues and expenditures and are derived 
from budget reports submitted by each school district to the PDE. These variables are 
displayed as a total amount, as a per pupil amount, and as a percent of total amount. 
The system was designed to include descriptors in as many different areas as possible, 
and at the same time limit the size of the database so it can be easily viewed and 
manipulated on a personal computer. 



One way the system displays the 
information is on four different screen 
formats: Studont, Revenue, Expense, 
and Professional Personnel as illustrated 
in Figure 1 . One set of four profile 
screens describes each of the 500 
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school districts. Each profile includes a Figure 1# student Profile, Average PA District 
HELP screen that provides variable definitions. Viewing time is controlled by the user 



permitting movement back and forth between the four profile screens and to move 
between districts. In addition to the school district profiles, a set of state summary 
profiles is included. Values displayed on the Pennsylvania screens are the average of 
all Pennsylvania districts. 



The options are presented in a 
popup menu displayed on the first screen 
of the main program as displayed in 
Figure 2. The options were selected to 
enable the user the opportunity to 



PEPS PC 

Pennsylvania School District Database 

One District's Profiles 
Profiles for Selected Districts 
Calculate Statistics 
Create a Customized Report 
Quit 



u^ 4 .,^ «. x ., , Figure 2. Main Menu, PEPS PC 

browse the profile screens and easily 

compare one district's information with others that are similar. In addition, options 
were designed to be easy to use and would enable information to be retrieved and 
displayed quickly, therefore simplicity of design was a priority. 

Selecting option one, "One District's Profiles", users can search for one specific 
school district and view that district's profile screens and also browse the profile 
screens of other cr all school districts. By selecting the order in which districts are 
displayed, users can compare districts that are similar. Districts can be ordered by: 
(a) school district name, (b) intermediate unit, (c) total expenses per pupil, (d) state aid 
ratio, (e) percent AFDC students, and (f) enrollment. 

Using options two, three, and four users can select districts sharing common 
characteristics. Using (a) "Profiles for Selected Districts", users view the profile 
screens of the districts that are similar; (b) "Calculate Statistics", users can calculate 
statistics on a specific field for selected or all districts; and (c) "Create a Report", users 
can list information in several fields for selected school districts. 
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Project Plan 

This software system was developed within the framework of a formative 
evaluation (George, 1992). It was targeted for use by Pennsylvania public school 
district administrators, though a broader population of potential users was also defined 
and included: (a) local and state government officials, (b) school district personnel, (c) 
community members, and (d) business representatives. 

The project plan was adapted from the Bertram and ChilJers (1974) model for 
evaluation of educational products. The major emphasis was on program development 
with a series of one-to-one tryouts, small group tryouts, and demonstrations. The 
system was first tested in a controlled setting and then operationally tested in an on- 
site setting. After each series of tryouts, the system was revised. Figure 3 illustrates 
the development sequence. 

Information was collected by interviewing and observing users during tryouts 
and demonstrations. In addition, on-site users completed a software evaluation 
questionnaire presented in Appendix A. The questionnaire was adapted from the 
Microcomputer Testing Software Review Form (Hsu & Nitko, 1983) developed for 
evaluating classroom testing software. Also consulted were the User's Evaluation 
Form (Hsu & Saddock, 1985), and forms suggested in Lawson's (1974) Formative 
Instructional Product Evaluation. The questionnaire generated information relative to 
users' (a) microcomputer and information use; (b) evaluation of PEPS PC options and 
the types of information included; and (c) ratings of installation, performance, content, 
screen display and format, documentation, and overall evaluation. Open-ended items 
provided an opportunity for suggestions and comments. 

In addition to information generated during tryouts, data describing the target 
population with respect to microcomputer use and information needs was generated 
using a survey of the targeted population, presented in Appendix B. This mailing was 
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sent to fifty-four Pennsylvania public school district leaders. This group included fifty- 
two school district superintendents, and two intermediate unit directors. This sample 
group was selected systematically with a random start drawn from the population of 
school district superintendents and intermediate unit directors. The sample population 
was first ordered by zip code, therefore stratifying geographically; selection of a 
starting point was random; end every 10th record was selected. This generated a 
10% sample. A total of 36 administrators responded to the survpy, 67% of the 
original sample. The following is a description of the development sequence. 



A preliminary version of PEPS PC was developed based on an earlier version 
developed by William Cooley and James Bernauer, interviews with the project director, 
William Cooley, and a review of the professional literature. A major goal for this 
version was to create an application that was user-friendly. When PEPS PC version 
1 was running smoothly, the system was field tested with a series of one-to-one and 
small group tryouts, demonstrations, and on-site tryouts in a controlled setting. The 
users were volunteers from the target population and population of potential users. 

One-to-one tryouts included eleven users, two public school district 
superintendents, five school district administrators, one legislative staff member from 
the House Education committee, three professors and one student from the 
Administrative and Policy Studies department. Each session took approximately forty- 
five minutes. The evaluator first demonstrated how to use the various options 
available, then observed and documented the users operation of the system. 

Small group tryouts were similar except that one group of six students from a 
dBASE IV class were observed by the evaluator. This version of PEPS PC was also 
demonstrated four times involving approximately thirty-four potential users. These 
tryouts and demonstrations were observed and documented by an evaluator. 
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tn addition, seven users operated the system on their personal computers either 
in their homes or at their workplace. This group included two school district 
superintendents and five school district administrators. All of the users had previously 
participated in one-to-one tryouts of the system. Feedback was collected using phone 
interviews. 

Though no major problems were noted in the program's performance and 
content this first series of field testing indicated the need to make the system easier 
to use, include additional types of information, and improve clarity of the screen 
displays and documentation. The most valuable feedback was generated by the on- 
site tryouts. The preliminary version of PEPS PC was written using the dBASE IV 
application and dependent on the dBASE IV application to run. As a result, on-site 
users had numerous problems trying to install and run the program. In addition, use 
of the program was severely limited due to the need for users to have the dBASE IV 
application. These problems indicated the need to recreate the program using an 
application that would produce a stand-alone version of PEPS PC and enable operation 
of the program without the need to own any additional software. This was done using 
Clipper 5.0 application, developed by the Nantucket Corporation (1990). Using the 
Clipper compiler, executable files were created resulting in a stand-alone version. This 
version was used in the next series of tryouts. 

When the system was revised, it was again tested using PEPS PC version 2. 
These tryouts differed from the field testing in two ways. First, during this stage an 
attempt was made to approximate the actual conditions in which this system will 
eventually be used. To do this, the evaluator conducted tryouts giving little or no 
assistance and also conducted on-site tryouts. Second, in addition to evaluator 
observations, a software evaluation questionnaire was completed by users. This 
questionnaire was described in the previous section. 
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The first series of tryouts were with a small group as in the field testing stage, 
except that this time they were given little help. Users were eleven students in a 
dBASE IV class. After some minor revisions, the system, PEPS PC version 3, was 
tested with on-site tryouts by seventeen users. This group included twelve school 
district administrators, two computer information experts, and one faculty member and 
two students from the school of education. The group of school district administrators 
was selected based on their response to Item 7 in the user's survey that asked if they 
were interested in trying out the PEPS PC school district database. The five other on- 
site users were volunteers. 

This group of on-site users was given no preliminary help. They were sent the 
PEPS PC database, the user's guide, the evaluation questionnaire, and instructions. 
The users were asked to operate the system for approximately three weeks and 
complete and return the questionnaire. The results of the user's survey and tryouts 
are described in the following section. 

Results 

The results of the informational survey of the target population are summarized 
in Table 1 . Most significantly, all respondents had access to a microcomputer system, 
and the microcomputer system available to 89% of the respondents was the IBM 
compatible system, the type necessary for operating the PEPS PC database system. 
A considerable number had easy access to a computer (75%, computer in their offices) 
and were frequent users, using the microcomputer for administrative purposes either 
daily or weekly (75%). Over 53% reported using. the computer daily. The most 
frequently used administrative applications were spreadsheet and word processing 
(86%, 86%), database (69%), communications (64%), and graphics (56%). The least 
frequently used type of software was data analysis (33%). 
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A vast majority of the respondents (81%) indicated an interest in using 
Pennsylvania school district information, most frequently for district assessment, 
budgeting, and contract negotiations (69%). Other reasons included, communicating 
with school board, the public, and teachers (53%, 53%, 39%). The results provided 
realistic guidelines for planning and development. 

The results of the tryouts and demonstrations provided constructive feedback 
regarding development of microcomputer applications. Overall users responses were 
extremely positive. They indicated a preference for a microcomputer database system 
that is user-friendly, which includes the following: (a) clearly written prompts, 
directions, and messages frequently displayed; (b) screen displays that are clear, 
simple, and easy to read and interpret; (c) frequent use of HELP screens that include 
descriptions of the options, directions for using them, and examples; (d) options that 
are menu-driven; and (e) documentation that includes clearly written directions and 
descriptions and numerous examples. 

Results indicated that errors were more frequent when the user was required 
to type in entries. Users also became impatient with a slow system and became 
frustrated if they wanted to exit before completing the option. Therefore, it was 
recommended to (a) have a relatively quick operation time; (b) use a picklist for 
selection of variables and options to make the operation easier, and reduce errors; and 
(c) provide an option to exit the screen display and return to the main menu to help to 
eliminate frustration. Training sessions, inservice, or tutorials may be necessary to 
encourage usage and limit frustrations. 

The on-sits users' evaluation of the PEPS PC system was overwhelming 
positive as indicated by their responses on the evaluation form. Users were asked to 
indicate the value of the options and the information, as well as to evaluate the 
installation procedure, performance, content, screen display and format, 
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Table 1. User's Survey: Summary of Results 
Respondents n = 36 



Position 


Freq 


% 


Total 


Freq 


% 


Superintendent 


16 


44% 


Male 


33 


92% 


Assistant Superintendent 


5 


14% 


Female 


3 


8% 


Business Manager 


5 


14% 








Coordinator 


5 


14% 


TOTAL 


36 




Principal 


3 


8% 








Other 


2 


6% 








TOTAL 


36 










Microcomputer System 












I npatinn 

L. \J V* O llUi 1 


Frecj 


% 


nocfnntinn 
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Freq 


/o 


Office 


27 


75% 


IBM Compatible 


32 


89% 


Central Office 


22 


61% 


Apple 


7 


19% 


District 


13 


36% 


Macintosh 


13 


36% 








Color Monitor 


22 


61% 


Microcomputer Use 


Freq 


% 


Mono Monitor 


11 


31% 


Daily 


19 


53% 


Hard disk drive 


27 


75% 


Weekly 


8 


22% 


Floppy (3 1/2") 


21 


58% 


Monthly 


- 


- 


Floppy (5 1/4") 


13 


36% 


Other 


7 


19% 


Printer 


28 


78% 


Missing 


2 


6% 








Application Software 












Own Admin Use 


Freq 


% 


Staff Admin Use 


Freq 


% 


Database 


25 


69% 


Database 


29 


81% 


Spreadsheet 


31 


86% 


Spreadsheet 


29 


81% 


Word processing 


31 


86% 


Word Processing 


35 


97% 


Graphics 


20 


56% 


Graphics 


22 


61% 


Data analysis 


12 


33% 


Data analysis 


16 


44% 


Communications 


23 


64% 


Communications 


23 


64% 


Other 






Other 


3 


8% 


Pennsylvania School District Information Use 








Reasons for Use 


Freq 


% 








District assessment 


25 


69% 








Budgeting 


25 


69% 








Contract negotiation 


25 


69% 








Communicate/teachers 


14 


39% 








Communicate/sch board 


19 


53% 








Communicate/public 


19 


53% 








Don't know 


9 


25% 








Other 


3 


8% 








Interest in Peps PC Trvout 














Freq 


% 








Yes 


14 


39% 








Maybe 


13 


36% 








No 


6 


17% 
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documentation and overall value. Of the seventeen on-site users, the majority (13) 
indicated that they used a microcomputer daily. On the average, they reported that 
the number of times they operated the PEPS PC system was approximately 2.6 times 
for a total accumulated time of two and one-half hours. 

The value of the options and information was determined based on a four point 
scale (1 =never useful, 2 = seldom useful, 3 = occasionally useful, and 4 = frequently 
useful). The results are summarized in Table 2. Users indicated that all four options 
were at least occasionally useful (means over 3.3), though, Options 1 and 2, One 
District's Profiles and Profiles for Selected Districts, were rated most frequently useful 
(mean = 3.9). Though all of the information was rated as at least occasionally useful, 
the revenue and expenditure information was rated most frequently useful (mean = 3.9, 
3.7). The coordinator information was considered least useful (mean = 2.7), but 
because this information was a subset of the professional personnel information and, 
therefore, affected the district values, it was decided to continue to include these data. 



Table 2. PEPS PC Evaluation Form: 

Summary of Options and Information 



Options 


Mean 


One District's Profiles 


3.9 


Profiles for Selected Districts 


3.9 


Calculate Statistics 


3.3 


Create a Customized Report 


3.4 


Information 


Mean 


Student Information 


3.5 


Administrator Information 


3.4 


Coordinator Information 


2.7 


Teacher Information 


3.4 


Expenditure Information 


3.9 


Revenue Information 


3.7 


Identification Information 


3.3 


n = 1 7 Rating Scale 


1-4 
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The remaining items of the evaluation form were rated on a five point scale 
with a five equal to strongly agree, the most favorable response. These are 
summarized in Table 3. Items that were stated negatively w;ere recoded so that a five 
represented the most favorable response. Because most of the users (76%) indicated 
that they used the microcomputer daily, and therefore, were experienced computer 
users, the responses were interpreted cautiously. It is likely that potential users will 
be less experienced with respect to computer use (George, 1992). Therefore, they 
may experience more problems, consequently, any problems reported, even if by a 
few, were considered seriously and indicated possible revisions. 

All users indicated that the installation procedure was easy to follow and error- 
free. Users' responses indicated that the system ran smoothly, directions were clear, 
sequence was logical, output alternatives were sufficient, and the system had few 
errors. Possible areas to be considered for revision related to training and to the 
availability of escape options. Six respondents felt that additional training would be 
necessary to operate the system effectively, it is likely that less experienced users 
would have even more problems. Therefore, planning additional training or inservice 
demonstrations was considered and recommended. Two respondents indicated the 
need for additional escape options. It is possible that these users were unaware that 
the escape key could be used at any time to exit. Further screen directions may be 
necessary. Another consideration is that using the escape key to abort is not 
sufficient. Including options to repeat and edit previous steps, especially when 
creating criteria, may be necessary. 

All users agreed that the information included was accurate, easy to interpret, 
and valuable for school district leaders; and that the screen displays were clear, easy 
to read, and the language was appropriate. They indicated that the guide was 
complete, logically organized, and easy to use. Four responses to item 48 indicated 
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the need for additional examples to illustrate how to effectively use the system. 
Again, because these responses were from experienced computer users, the use of 
additional examples may be helpful in a future version. Users indicated that the 
system was an excellent tool for district level use and strongly recommended its use. 

Some respondents felt that the information was not current (the tested version 
included 1 989-90 data and was tested in 1 991 ). The information is obtained from the 
Pennsylvania Department of Education, and will always be at least one year behind. 
Because this database is presented as a means for school districts to view and 
compare their district with others in the state, the apparent problem about currency 
of data may not be substantial. Cooley (1990) reports that variations in ranking and 
comparisons between school districts remain relatively constant across years. This 
would mean, that despite the change in total values the relativity of the values remains 
stable. In addition, the database will be updated annually. Though the currency of the 
data can not be changed, revising the documentation to include an explanation of the 
stability of the comparisons and rankings, as well as information about the plans for 
annual updates may be beneficial. 

The evaluation form also included four open-ended items. These items gave 
users the opportunity to include comments or suggestions for revisions and to indicate 
elements that caused frustration. Many of the responses were positive and included 
the following: 

I have shared this database with other administrators of our district and 
they are very positive about it. 

The program is very impressive. 

In response to a board member's statement that this district had the 
highest number of administrators per pupil, I was able check using this 
system and show there was no truth to that statement. 

It is ... an excellent, quick way to compare districts-statewide. This 
is most useful at negotiation time. 
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Table 3. Operational Testing: Summary of PEPS PC Evaluation Form 
On-site testing: n = 1 7 



Proaram Performance 




Users 


Freq 


Items Rating Mean 


Supt, Director 


2 






District Admin 


9 


Program runs smoothly 


4.6 


Other 


6 


Directions, clearly stated 


4.5 


Total n 


17 


Error handling, sufficient 


3.8 






Program sequence, logical 


4.3 


Frea PC Use 


Freq 


Additional training, not necessary 


3.3 




Information, retrieved quickly 


4.2 


Daily 


13 


Output alternatives, sufficient 


3.8 


Weekly 


3 


Escape options, sufficient 


4.1 


Other 


1 


Program operation, not time-consuming 


4.2 






Add 'HELP' screens; not necessary 


3.2 


PEPS PC Use 


Mean 


Proaram Content 




Number of Times 


2.6 






Hours 


2.5 


Information is accurate 


4.2 






Information is current 


3.9 


Information Use 


Freq 


Information is easy to interpret 


4.2 




Information is valuable for admin, use 


4.5 


District Assess. 


13 






Budget 


9 


r^icntsv/ f-/*"irm;*+ 

oi^iccm L/iouiaYj rormai 




Negotiations 


6 






Communicate with: 




Displays are clear, easy to read 


4.4 


Prof. Personnel 


7 


Language is appropriate 


4.3 


School Board 


9 


Di^nlav** nnt pnnfiiQlnn 




Public 


8 


Displays, free of errors 


4.3 






User's Guide 








Guide includes essential information 


4.4 






Dictionary describes variables clearly 


4.0 






Guide is logically organized 


4.2 






Information is located easily 


4.2 






No additional examples necessary 


3.0 










Rating 




Overall Evaluation 




5 = Strongly Agree 








4 = Agree 




PEPS PC is an excellent tool for admin. 


4.5 


3 = Neutral 




PEPS PC is highly recommended 


4.6 


2 = Disagree 




PEPS PC needs no revisions 


3.1 


1 = Strongly Disagree 
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Other users' responses provided valuable feedback with respect to system 
revisions. Some of the suggestions indicated the following additions: (a) options to 
revise search criteria, (b) additional escape options, (c) better error handling procedure, 
(d) additional information, (e) trends over time, and (f) options to edit criteria previously 
defined. These suggestions will be considered for future versions. 

These results were considered with respect to plans for revisions and for future 
development. Time constraints and funding limited the revisions to only those 
determined absolutely necessary to include in the distribution version that was be used 
in dissemination. Other revisions will be considered for future versions. Revisions, 
therefore, were limited to improving clarity of screen displays and improving and 
revising the documentation. When these revisions were completed, the distribution 
version was ready for dissemination. 

Based on the results of the survey of the targeted population (district leaders 
and administrators) that indicated that over 75% of those responding were either 
definitely or possibly interested in using the PEPS PC microcomputer system and 
approximately 89% of the districts had the necessary hardware, the dissemination 
took place in two phases. PEPS PC was first made available to a sample of the 
targeted population. Plans for future dissemination will be based on results of the 
earlier phase. 

Summary and Conclusion 
The purpose of this study was to design, develop, and evaluate a 
microcomputer database system within the framework of a formative evaluation. The 
school district database was targeted for use by public school district administrators. 
It is argued that timely, relevant, and easily accessed information can help support 
district level decisionmaking (Cooley, 1990; Crawford, 1985; Cunningham, 1982; 
Turbin, Gisher, & Altman, 1988). In an attempt to meet these needs, this database 
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contains current information that is descriptive of the 500 school districts in 
Pennsylvania and describes students, professional personnel, and financial operations. 
It was designed to be user-friendly, with options to browse, search, define queries, 
calculate simple statistics, and create a custom report. 

Tne model selected for this formative evaluation was the Bertram and Childers 
model (1974). The model was adapted to meet the specific needs of software 
development and emphasized development within a series of field testing. The design 
and development strategy was the emphasis of this study and required the major 
portion of the time. This component included fibid testing and operational testing 
stages. During tryouts, the evaluator recorded problems, comments, and suggestions 
of the users. Testing included tryouts in a controlled situation and also included on- 
site tryouts. As an additional means of collecting feedback, the operational testing 
included completion of a software evaluation questionnaire. Revisions were based on 
feedback. This process continued until completion of a distribution version. The PEPS 
PC statewide distribution will be based on the results of the first phase of the 
dissemination. 

The model was particularly well suited to this project. Because the system was 
a microcomputer application, data collection activities focused on user tryouts. By 
observing and documenting usage, the evaluator was able to note problems associated 
with the hardware as well as those with the application. In addition, the on-site 
tryouts generated information regarding performance relative to the actual use. 

The evaluator felt that the part of this study that produced the most useful 
information was the field testing with one-to-one and small group tryouts using 
evaluator interviews and documented observations. This stage was most valuable for 
two reasons. First, the users included both experienced and inexperienced computer 
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users. Second, the evaluator was able to learn a great deal about the operating 
problems by observing the user and responding to his/her questions. 

During on-site tryouts, the method of selection of the users and the types of 
data collection activities designed were somewhat different from the earlier field 
testing tryouts and therefore limited the feedback generated. Because the sample 
group in the operational testing was selected from district administrators who 
volunteered, the sample group consisted of experienced computer users, therefore, the 
results had to be interpreted cautiously. In addition, data were collected using the 
PEPS PC evaluation questionnaire. Responses were, therefore, limited by the items on 
the form. Though the evaluator felt that the operational testing was a useful and 
necessary part of this study, generating information with respect to application use on 
many and different computers and without the aid of an evaluator, it may have been 
more useful if data collection activities had included documented observations and/or 
interviews in addition to the users' responses on the evaluation form. 

Providing school district data in a form that includes a means of comparing 
districts and in an easily accessed database can have implications for school district 
administrators and decisionmakers as well as community members, educators, and 
outside participants. The availability of easily accessed school district information can 
provide school district administrators with the foundation for feedback regarding the 
condition of the schools and the school district with respect to other districts. The 
availability of a relevant database can provide the district administrators with a 
powerful tool for district assessment, negotiations, planning and decisionmaking. In 
addition, the same information can provide the means for participants outside of the 
school district to learn of the condition of the school district with respect to other 
districts. It can also provide outside participants with an overview of how the school 
districts are performing. Making information open and available may raise questions 
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and/or stimulate debate. The consequences of providing this information to 
participants within the school district, as well as those who are outside, can be 
explored in further study as the database becomes more widely used. 

The possibilities for future research are exciting. With respect to software 
development, the database can be considered a work in progress. Feedback from 
users can continue to be used to refine this database. In addition, feedback from 
district administrators can provide indicators regarding how and what information is 
desirable. Observing the consequences of making such information available to the 
public can also provide valuable feedback describing how information can effect 
change in the public school district. In summary, this study provides a basis for 
learning about how information impacts school districts and the way they operate. 
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APPENDIX A 



User's Survey 
PEPS PC Pennsylvania School District Database 



Nam* Position^ 

Work Address 



Phone 



Microcomputer and Information Osage (Check all relevant items) 

1. Do you have access to a microcomputer 

In your office? 

In the central office? 



Elsewhere in the district? 
None available? 



2. Describe the microcomputer system that you use or is available for your use. 
IBM compatible microcomputer 



_Apple microcomputer 
^Macintosh system 
_Color monitor 
_Mono monitor 
Hard disk drive 
.Floppy diskette (3 1/2") 
^Floppy diskette (5 1/4") 
J?r inter 
_Other (specify) 



3. How often do you personally use the microcomputer? 

Daily Weekly Monthly Other (specify)^ 



Please check the types of applications software used in your district. 

Personal Use Staff Use 
Database Database 



^Spreadsheet Spreadsheet 

_Word processing Word processing 

jCraphics Graphics 

_Data analysis Data analysis 

jCommuni ca t i on s Communi cat ion s 

_Other (specify) O ther (specify) _ 
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5. Please indicate the extent to which you might um information describing 
other Pennsylvania school districts (as would be available with the PEPS PC 
database described previously), if it were available. 

Daily Weekly Monthly Don't Know 



6. How might you use Pennsylvania school district information? 

District assessment 

Budgeting 

Contract negotiations 

Communicating with teachers 

Communicating with school board members 

Communicating with public 

Don' t know 

Other (describe) 



PEPS PC School District Database 

7. Would you be interested in trying out the PEPS PC Pennsylvania School 
District Database (at your location)? 

Y **' 1 would like to try out the system using my microcomputer. 

1 »*Y interested, first I would like to learn more about it. 

No, I am not interested in trying out the system. 
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APPENDIX B 

PEPS PC Evaluation Form 
PEPS PC Pennsylvania School District Database 



Name 



Position 



Part One DIRECTIONS: Please complete this form. Most items require only 
a check mark ( ) . Some may require more than one response, if the it em 
is not applicable, leave it blank. 



1. How often do you personally use the microcomputer? 
Daily W eekly Monthly Other (specify) 



How often have you personally used the PEPS PC Pennsylvania School 
District Database during the last month (or weeks)? 



2 . Number of Times 

One to two times 

Three to four times 

Five to seven times 

Eight to ten times 

More than ten times 



3. Total Amount of Time 

Less than one hour 

One hour to two hours 

Two hours to three hours 

Three hours to four hours 



More than four hours 



Now that you have had a chance to use PEPS PC, how do you expect to 
use Pennsylvania school district information? (Check all relevant 
items) 



4. 

5. 
6. 
7. 
8. 
9. 
10, 



^District assessment 
^Budgeting 

^Contract negotiations 

^Communicating with district personnel 
^Communicating with school board members 
^Communicating with public 
_pther (describe)_ 
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Part Two DIRECTIONS : Indicate the usefulness of the options and 
information included in the PEPS PC system by circling the appropriate 
numeral . 



4=Frequently Useful 
3=Occasionally Useful 
2=Seldom Useful 
l=Never Useful 



When using the PEPS PC Pennsylvania School District Database, how 
useful were the following options? 

Freq. Occas . Seldom Never 
Useful Useful Useful Useful 



11. One District's Profiles 4 3 2 1 

12. Profiles for Selected Districts 4 3 2 1 

13. Calculate Statistics 4 3 2 1 

14. Create a Customized Report 4 3 2 1 



When using the PEPS PC Pennsylvania School District Database, how 
useful were the following kinds of information? 

Freq. Occas. Seldom Kever 

Useful Useful Useful Useful 



15 . Student information 

16. Administrator information 

17 . Coordinator information 

18 . Teacher information 

19. Expenditure information 

20. Revenue variables 

21. Identification variables 



3 
3 
3 
3 
3 
3 
3 



2 
2 
2 
2 
2 
2 
2 



1 
1 
1 
1 
1 
1 
1 



22 . What additional options or kinds of information would you find useful? 
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Part 3: PEPS PC Pennsylvania School District Database Evaluation 



DIRECTIONS : Read each statement and decide the extent to which you agree 
or disagree. Circle the appropriate numeral to indicate your response. 

5=Strongly Agree (SA) 
4= Agree (A) 
3=Neutral (N) 
2=Disagree (D) 
l=Strongly Disagree (SD) 



Installation SA A 

23. The installation procedure was easy to follow. 5 4 

24. Installation was error-free. 5 4 

25 . Installing the program required additional 

help . 5 4 



N 
3 
3 



D 
2 
2 



B 
1 
1 



Program Perfonuance SA A 

26. The program runs smoothly with few errors. 5 4 

27. The directions are clearly stated. 5 4 

28. The error handling procedure is sufficient. 5 4 

29. The program sequence is logical. 5 4 

30 . Additional training is necessary to use the 
program effectively. 5 4 

31. Information is retrieved and displayed 

quickly. 5 4 

32 . Output alternatives are sufficient . 5 4 

33. The program has sufficient "Escape" options 

to permit exit at any time. 5 4 

34 . Operating the program is t ime -cons umi ng and 
tedious . 5 4 

35. Additional "Help" screens are necessary in 

order to operate the program effectively. 5 4 



N D B 
3 2 1 
3 



3 
3 



3 
3 

3 

3 

3 



2 1 

2 1 

2 1 

2 1 

2 1 

2 1 

2 1 

2 1 

2 1 



Program Content SA A 

36. The information is accurate. 5 4 

37. The information is reasonably current. 5 4 

38. The information is presented in a form that 

is easy to interpret . 5 4 

39. The information has little value for 

school district leaders. 5 4 



N D B 

3 2 1 

3 2 1 

3 2 1 

3 2 1 
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Screen Display and Format 

40. Screen displays are clear and easy to read. 

41. The language used is appropriate. 

42 . Screen displays contain too mv.ch confusing or 
extraneous information . 

43. Screen displays are free from grammatical, 
spelling, or typographical errors. 



SA 

5 

5 



A 
4 
4 



N D B 
3 2 1 
3 



2 1 
2 1 
2 1 



User' s Guide SA 

44. The User's Guide includes essential information 
needed to use the program effectively. 5 

45 . The database dictionary describes variables 
clearly and completely. 5 

46. The User's Guide is logically organized. 5 

47. The information is located easily. 5 

48. Additional examples are necessary to illustrate 
how the program can be most effectively used. 5 



Overall Evaluation SA A 

49. PEPS PC system is an excellent tool for 

district level decision making and planning. 5 4 

50. PEPS PC system is highly recommended. 5 4 

51. The PEPS PC system requires additional 

revisions . 5 a 



N D B 



2 1 



3 2 1 

3 2 1 

3 2 1 

3 2 1 

N D B 



2 1 
2 1 

2 1 



52 . Please indicate ways to improve the operation or usefulness of this 
system. 



53. What additional indexes (order options) would you find useful? 



54. Please indicate elements of the system that caused frustration. 
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